Method and apparatus for virtualizing a computer accessory

ABSTRACT

A system that incorporates the subject disclosure may include, for example, device may perform operations for receiving a signal from an operational space associated with an accessory device, determining hand feature information according to the signal, accessing device location information associated with the computing accessory device, generating hand-device interaction information according to the hand feature information and device location information associated with the accessory device, and transmitting the hand-device interaction information to a virtual reality system, wherein the virtual reality system generates, according to the hand-device interaction information, a virtual hand and a virtual accessory device in a virtual reality image. Additional embodiments are disclosed.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/867,901 filed on Jan. 11, 2018. All sections of the aforementionedapplication are incorporated herein by reference in their entirety.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to a method and apparatus forvirtualizing a computer accessory.

BACKGROUND

It is common today for gamers to utilize more than one gaming accessory.This is especially true of gamers who play on-line games or competitivegames in a team or individual configuration. Gamers can have at theirdisposal accessories such as a keyboard, a general purpose gaming pad, amouse, a gaming console controller, a headset to communicate with otherplayers, a joystick, a computer console, or other common gamingaccessories.

A gamer can frequently use a combination of these accessories in asingle game (e.g., headset, a keyboard, and mouse). Efficient managementand utilization of these accessories can frequently impact a gamer'sability to compete. Accessory management can have utility in otherdisciplines which may not relate to gaming applications. Efficient useof accessories in these other disciplines can be important to otherusers.

Virtual Reality (VR) products have increased in availability andconsumer acceptance. VR applications, typically in the gaming andentertainment space but also for development and training, have alteredthe landscape for computer input and output devices. Consumers demandinput, output, and accessory devices that compliment VR capabilities andexpectations.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 depicts an illustrative embodiment of a Graphical User Interface(GUI) generated by an Accessory Management Software (AMS) applicationaccording to the present disclosure;

FIGS. 2-3 depict illustrative embodiments for communicatively coupling agaming controller to a computing device;

FIG. 4 depicts an illustrative embodiment of a communication device;

FIG. 5 depicts an illustrative embodiment of a first method utilized inthe subject disclosure;

FIG. 6 depicts an illustrative embodiment of a second method utilized inthe subject disclosure;

FIG. 7A depicts an illustrative embodiment of a third method utilized inthe subject disclosure;

FIG. 7B depicts an illustrative embodiment of a fourth method utilizedin the subject disclosure;

FIG. 8 depicts an illustrative embodiment of a virtualized accessorydevice;

FIGS. 9A and 9B depict illustrative embodiments of a system including avirtualized accessory device;

FIG. 10 depicts an illustrative embodiment of a virtualized accessorydevice capturing information from a user's hands;

FIG. 11 depicts an illustrative embodiment of generating a virtualaccessory device and virtual hands for display in a virtual application;

FIGS. 12A and 12B depict illustrative embodiments of virtualapplications incorporating a virtual accessory device and virtual hands;

FIG. 13 depicts an illustrative embodiment of a system operating atleast in part according to the methods of FIGS. 5-7B;

FIG. 14 depicts an illustrative embodiment of a communication flowdiagram utilized by the system of FIG. 13 ; and

FIG. 15 depicts an illustrative diagrammatic representation of a machinein the form of a computer system within which a set of instructions,when executed, may cause the machine to perform any one or more of themethodologies disclosed herein.

DETAILED DESCRIPTION

The subject disclosure describes, among other things, illustrativeembodiments for virtualizing an accessory device. In one or moreembodiments, a system can receive infrared light from an operationalspace associated with the accessory device. Based on the receivedinfrared light, the system can determine hand feature information fromthe operation accessory device. The system can map the hand featureinformation and device location information to generate a set ofhand-device information that is associated with the operational space.The system can then transmit the hand-device information to a virtualreality operating system, which can generate virtual versions of theaccessory device and operator hand (or hands) that can be displayed in avirtual environment of a virtual reality application. Other embodimentsare described in the subject disclosure.

One embodiment of the subject disclosure includes a method, performed bya processing system including a processor, where the method can includeprojecting first infrared light into an operational space associatedwith an accessory device and receiving second infrared light from theoperational space associated with the accessory device. The method canalso include generating a hand image according to a first portion of thesecond infrared light that is received and, in turn, determining handfeature information from the hand image according to device locationinformation associated with the accessory device. The method can furtherinclude mapping the hand feature information and the device locationinformation to generate hand-device interaction information associatedwith the operational space. The method can also include modifying thehand-device interaction information according to input data associatedwith the accessory device to generate modified hand-device interactioninformation. The method can further include transmitting the modifiedhand-device interaction information to a virtual reality operatingsystem. The modified hand-device interaction information can be providedto a virtual reality application by the virtual reality operatingsystem. The virtual reality application can superimpose a virtual handand a virtual accessory device onto a virtual reality image according tothe modified hand-device interaction information.

One embodiment of the subject disclosure includes a machine-readablestorage medium, including instructions, where, responsive to executingthe instructions, a processing system including a processor can performsoperations including receiving image information from an operationalspace associated with an accessory device and, in turn, generating ahand image from a first portion of the image information that isreceived. The operations can also include determining hand featureinformation from the hand image according to device location informationassociated with the accessory device. The operations can further includegenerating hand-device interaction information associated with theoperational space according to the hand feature information and thedevice location information. The operations can also includetransmitting the hand-device interaction information to a virtualreality operating system. The hand-device interaction information can beprovided to a virtual reality application by the virtual realityoperating system. The virtual reality application can superimpose avirtual hand and a virtual accessory device onto a virtual reality imageaccording to the hand-device interaction information.

One embodiment of the subject disclosure includes a device including amemory to store instructions and a processing system including aprocessor coupled to the memory, wherein responsive to executing theinstructions, the processor performs operations including receiving asignal from an operational space associated with an accessory deviceand, in turn, determining hand feature information according to thesignal. The operations can also include generating hand-deviceinteraction information according to the hand feature information anddevice location information associated with the accessory device. Theoperations can further include transmitting the hand-device interactioninformation to a virtual reality system. The virtual reality system cangenerate, according to the hand-device interaction information, avirtual hand and a virtual accessory device in a virtual reality image.

FIG. 1 depicts an illustrative embodiment of a Graphical User Interface(GUI) generated by an Accessory Management Software (AMS) applicationaccording to the present disclosure. The AMS application can be executedby a computing device such as a desktop computer, a laptop computer, atablet, a server, a mainframe computer, a gaming console, a gamingaccessory, an accessory device, or any combination or portions thereof.The AMS application can also be executed by portable computing devicessuch as a cellular phone, a personal digital assistant, or a mediaplayer. The AMS application can be executed by any device with suitablecomputing and communication resources.

FIG. 2 illustrates a number of embodiments for utilizing a gamingcontroller 115 with a computing device 206 in the form of a gamingconsole. In the illustration of FIG. 2 , the gaming controller 115 canbe communicatively coupled to the gaming console 206 with a tetheredcable interface 202 such as a USB or proprietary cable, or a wirelessinterface 204 such as WiFi, Bluetooth, ZigBee, or a proprietary wirelesscommunications protocol. The cable interface 202 provides a means forcommunication that may be less susceptible to electromagneticinterference. It will be appreciated that the gaming controller 115 mayfurther include a keyboard device 108 or a headset 114 (with or withouta microphone not shown) utilized by a gamer to communicate withteammates and/or to listen to game sounds in high fidelity. In theillustration of FIG. 2 , the AMS application can in whole or in part beexecuted by the gaming controller 115, the gaming console 206, or acombination thereof.

FIG. 3 illustrates a number of other embodiments for utilizing a gamingcontroller 115 or 112 with a computing device 206. In this embodiment,the gaming controller 115 comprises a mouse and the computing device 206comprises a computer. The gaming controller 115 can be tethered to thecomputing device 206 by a cable interface 202 (e.g., USB cable orproprietary cable) or a wireless interface 204. The cable interface 202provides a means for communication that may be less susceptible toelectromagnetic interference. It will be appreciated that the gamingcontroller 115 may further include a keyboard device 108 or a headset(with or without a microphone not shown) utilized by a gamer tocommunicate with teammates and/or to listen to game sounds in highfidelity. In the illustration of FIG. 3 , the AMS application can inwhole or in part be executed by the gaming controller 115, the gamingconsole 206, or a combination thereof.

For illustration purposes, the terms gaming console 206 and computer 206will be used hence forth interchangeably with the term computing device206 with an understanding that a computing device 206 may represent anumber of other devices such as a server, a tablet, a smart phone, andso on. Accordingly, a computing device 206 can represent any device withsuitable computing resources to perform the methods described in thesubject disclosure.

FIG. 4 depicts an illustrative embodiment of a communication device 400.Communication device 400 can serve in whole or in part as anillustrative embodiment of devices described in the subject disclosure.The communication device 400 can comprise a wireline and/or wirelesstransceiver 402 (herein transceiver 402), a user interface (UI) 404, apower supply 414, a proximity sensor 416, a motion sensor 418, anorientation sensor 420, and a controller 406 for managing operationsthereof. The transceiver 402 can support short-range or long-rangewireless access technologies such as Bluetooth, WiFi, Digital EnhancedCordless Telecommunications (DECT), or cellular communicationtechnologies, just to mention a few. Cellular technologies can include,for example, CDMA-1X, UMTS/HSDPA, GSM/GPRS, TDMA/EDGE, EV/DO, WiMAX,software defined radio (SDR), Long Term Evolution (LTE), as well asother next generation wireless communication technologies as they arise.The transceiver 402 can also be adapted to support circuit-switchedwireline access technologies (such as PSTN), packet-switched wirelineaccess technologies (such as TCP/IP, VoIP, etc.), and combinationsthereof.

The UI 404 can include a depressible or touch-sensitive keypad 408coupled to a navigation mechanism such as a roller ball, a joystick, amouse, or a navigation disk for manipulating operations of thecommunication device 400. The keypad 408 can be an integral part of ahousing assembly of the communication device 400 or an independentdevice operably coupled thereto by a tethered wireline interface (suchas a USB cable) or a wireless interface supporting for exampleBluetooth. The keypad 408 can represent a numeric keypad, and/or aQWERTY keypad with alphanumeric keys. The UI 404 can further include adisplay 410 such as monochrome or color LCD (Liquid Crystal Display),OLED (Organic Light Emitting Diode) or other suitable display technologyfor conveying images to an end user of the communication device 400.

In an embodiment where the display 410 utilizes touch-sensitivetechnology, a portion or all of the keypad 408 can be presented by wayof the display 410 with navigation features. As a touch screen display,the communication device 400 can be adapted to present a user interfacewith graphical user interface (GUI) elements that can be selected by auser with a touch of a finger. The touch screen display 410 can beequipped with capacitive, resistive or other forms of sensing technologyto detect how much surface area of a user's finger has been placed on aportion of the touch screen display. This sensing information can beused to control the manipulation of the GUI elements.

The UI 404 can also include an audio system 412 that utilizes commonaudio technology for conveying low volume audio (such as audio heardonly in the proximity of a human ear) and high volume audio (such asspeakerphone for hands free operation, stereo or surround sound system).The audio system 412 can further include a microphone for receivingaudible signals of an end user. The audio system 412 can also be usedfor voice recognition applications. The UI 404 can further include animage sensor 413 such as a charged coupled device (CCD) camera forcapturing still or moving images and performing image recognitiontherefrom.

The power supply 414 can utilize common power management technologiessuch as replaceable or rechargeable batteries, supply regulationtechnologies, and charging system technologies for supplying energy tothe components of the communication device 400 to facilitate long-rangeor short-range portable applications. Alternatively, the charging systemcan utilize external power sources such as DC power supplied over aphysical interface such as a USB port or by way of a power cord attachedto a transformer that converts AC to DC power.

The proximity sensor 416 can utilize proximity sensing technology suchas an electromagnetic sensor, a capacitive sensor, an inductive sensor,an image sensor or combinations thereof. The motion sensor 418 canutilize motion sensing technology such as an accelerometer, a gyroscope,or other suitable motion sensing technology to detect movement of thecommunication device 400 in three-dimensional space. The orientationsensor 420 can utilize orientation sensing technology such as amagnetometer to detect the orientation of the communication device 400(North, South, West, East, combined orientations thereof in degrees,minutes, or other suitable orientation metrics).

The communication device 400 can use the transceiver 402 to alsodetermine a proximity to a cellular, WiFi, Bluetooth, or other wirelessaccess points by common sensing techniques such as utilizing a receivedsignal strength indicator (RSSI) and/or a signal time of arrival (TOA)or time of flight (TOF). The controller 406 can utilize computingtechnologies such as a microprocessor, a digital signal processor (DSP),and/or a video processor with associated storage memory such as Flash,ROM, RAM, SRAM, DRAM or other storage technologies.

The communication device 400 can serve as an accessory device for acomputer system. For example, the communication device 400 can be akeyboard device 108, a mouse device 110, or a game controller 115. Inone or more embodiments, the communication device 400 can providevirtualization capabilities. The communication device 400 can facilitatedetection of user interactions with the accessory device in real space,where these user interactions can be translated to a VR domain forpresentation in the VR space.

The communication device 400 can include a light source 426. In oneembodiment, the light source 426 can provide light to an operationalarea associated with a communication device 400. For example, light canbe provided to an area associated with user interaction or operation ofan accessory device, such as a keyboard device 108. The light source 426generate and/or project infrared light or any other lighting frequencythat can be used to illuminate, detect, or expose physical positionsand/or interactions between a user and the keyboard device 108. In oneembodiment, the light source 426 can provide light to illuminate and/orreflect from fingers, finger tips, and other features of a user's handor hands. The light source 426 can provide light to illuminated and/orreflect from features of the accessory device to provide position and/ororientation information for accessory device. The light source 426 canbe included in the communication device 400 or can be included inanother communication device 400.

The communication device 400 can include a camera 428. In oneembodiment, the camera 428 can be capture light from an operational areaassociated with of a communication device 400. For example, the camera428 can capture light emanating from or reflected off of a communicationdevice 400, a user feature (such as a user's hand or fingertip, and/oran accessory device. In one embodiment, the camera 428 can receive lightin or near the infrared spectrum. Infrared light is typically generatedassociated with the transmission of heat. So, for example, many livingthings, such as human beings, will give off heat in the form of infraredradiation. This infrared radiation can be captured as infrared light viaa camera 428. Therefore, if a user is interacting with an accessorydevice, such as a keyboard device 108, using her hand(s), then thecamera 428 can capture infrared light that is emitted from her hand(s).In one embodiment, the camera 426 can be designed (or the incoming lightcan be sufficiently filtered) to accept light in or near the infraredspectrum while rejecting or ignoring light in other parts of the lightspectrum. In this way, the camera 426 can lock in on the infrared lightwhile rejecting other light frequency components. In one embodiment, thecamera 428 can capture light that is reflected from user features, suchas the user's hands, and/or that is emitted or reflected from theaccessory device. The camera 426 can capture features of the user andfeatures of the accessory device.

In one embodiment, the images captured by the camera 426 can be used todetermine location information for the features of a user's hand(s) asthose features interact with the accessory device. For example, thecapture images can provide real time locations of fingers, fingertips,hands, and so forth, which can be used to determine and/or confirmkeystrokes. The capture images can also provide real time locationand/or orientation information for the accessory device. For example,light can be capture as it is reflected from and/or emitted fromorientation markings on the accessory device. Real time information onthe location of the user features and the location of the accessorydevice can be provided to a VR operating system and/or VR application.The VR operating system and/or application can convert the real timeinformation on user-device interactions into the VR space where, forexample, virtualized versions of the user's hand(s) and the accessorydevice can be generated and presented in the VR space. In this way, theuser experience in the VR space can be enriched by the visual presenceof her hand(s) and the compute accessory device and the user's abilityto operate the accessory device will be enhanced. In short, the user cansee in VR (e.g., through VR goggles) what her hands are doing in realspace with the accessory device.

The communication device 400 as described herein can operate with moreor less components described in FIG. 4 to accommodate the implementationof devices described by the subject disclosure. These variantembodiments are contemplated by the subject disclosure.

FIGS. 5-7A depict methods 500-700 describing illustrative embodiments ofthe AMS application. Method 500 can begin with step 502 in which the AMSapplication is invoked in a computing device. The computing device canbe a remote server (not shown), the gaming console 206 or computer 206of FIGS. 2-3 , or any other computing device with suitable computingresources. The invocation step can result from a user selection of theAMS application from a menu or iconic symbol presented by the computingdevice 206, or when a user communicatively couples a gaming controller115 or other form of accessory device with the computing device 206. Instep 504, the AMS application can detect by way of software drivers inan operating system (OS) of the computing device 206 a plurality ofoperationally distinct accessories communicatively coupled to thecomputing device 206. The accessories can be coupled to the computingdevice 206 by a tethered interface (e.g., USB cable), a wirelessinterface (e.g., Bluetooth or Wireless Fidelity—WiFi), or combinationsthereof.

In the present context, an accessory can represent any type of devicewhich can be communicatively coupled to the computing device 206 (orwhich can be an integral part of the computing device) and which cancontrol aspects of the OS and/or a software application operating fromthe computing device 206. An accessory can represent for example akeyboard, a touch screen display, a gaming pad, a gaming controller, amouse, a joystick, a microphone, or a headset with a microphone—just tomention a few.

In step 506, the AMS application presents a GUI 101 such as depicted inFIG. 1 depicting operationally distinct accessories such as a keyboard108, and a gaming controller 115. The GUI 101 presents the accessories108-116 in a scrollable section 117. One or more accessories can beselected by a user with a mouse pointer. In this illustration, thekeyboard 108 and the gaming controller 115 were selected forcustomization. Upon selecting the keyboard 108 and the gaming controller115 from the scrollable window of section 117, the AMS applicationpresents the keyboard 108 and the gaming controller 115 in split windows118, 120, respectively, to assist the user during the customizationprocess.

In step 508, the AMS application can be programmed to detect auser-selection of a particular software application such as a videogame. This step can be the result of the user entering in a Quick Searchfield 160 the name of a gaming application (e.g., World of Warcraft™ orWoW). Upon identifying a gaming application, the AMS application canretrieve in step 510 from a remote or local database gaming applicationactions which can be presented in a scrollable section 139 of the GUIrepresented as “Actions” 130. The actions can be tactical actions 132,communication actions 134, menu actions 136, and movement actions 138which can be used to invoke and manage features of the gamingapplication.

The actions presented descriptively in section 130 of the GUI canrepresent a sequence of accessory input functions which a user canstimulate by button depressions, navigation or speech. For example,depressing the left button on the mouse 110 can represent the tacticalaction “Reload”, while the simultaneous keyboard depressions “Ctrl A”can represent the tactical action “Melee Attack”. For ease of use, the“Actions” 130 section of the GUI is presented descriptively rather thanby a description of the input function(s) of a particular accessory.

Any one of the Actions 130 can be associated with one or more inputfunctions of the accessories being customized in windows 118 and 120 byway of a drag and drop action or other customization options. Forinstance, a user can select a “Melee Attack” by placing a mouse pointer133 over an iconic symbol associated with this action. Upon doing so,the symbol can be highlighted to indicate to the user that the icon isselectable. At this point, the user can select the icon by holding theleft mouse button and drag the symbol to any of the input functions(e.g., buttons) of the keyboard 108 or selectable options of the gamingcontroller 115 to make an association with an input function of one ofthese accessories. Actions of one accessory can also be associated withanother accessory that is of a different category. For example, keydepressions “Ctrl A” of the keyboard 108 can be associated with one ofthe buttons of the gaming controller 115 (e.g., the left button 119).

In one embodiment, a Melee Attack action can be associated by draggingthis action to either the left button 119 or right button 120 of thegaming controller 115. Thus, when the selected button is depressed, thestimulus signal that is generated by the selected button of the gamingcontroller 115 can be substituted by the AMS application with the MeleeAttack action. In another embodiment, the AMS application can beconfigured so that the Melee Action can be associated with a combinationof key button presses (e.g., simultaneous depression of the left andright buttons 119, 121, or a sequence of button depressions: two rapidleft button depressions followed by a right button depression).

In yet another embodiment, the AMS application can be configured so thatthe Melee Action can be associated with movement of the gamingcontroller 115 such as, for example, rapid movement or shaking of thegaming controller 115. In a further embodiment, the AMS application canbe adapted to make associations with two dimensional or threedimensional movements of the gaming controller 115 according to a gamingvenue state. For example, suppose the player's avatar enters a fighterjet. In this gaming venue state, moving the left navigation knob forwardcan be associated by the AMS application with controlling the throttleof the jet engines. Rapidly moving the gaming controller 115 downwardcan represent release of munitions such as a bomb.

In a gaming venue state where the gamer's avatar has entered a building,lifting of the gaming controller 115 above a first displacementthreshold can be associated with a rapid movement of the avatar up onefloor. A second displacement threshold can be associated with a rapidmovement of the avatar down one floor—the opposite of the firstdisplacement threshold. Alternatively, the second displacement thresholdcould be associated with a different action such as jumping betweenbuildings when the avatar is on the roof of a building.

The AMS application can monitor gaming venue states by analyzingcaptured images produced by the gaming application (e.g., one or morestill images of a tank, or a video of an avatar entering a tank), and/orby receiving messages from the gaming application by way of anapplication programming interface (API) thereby enabling the AMSapplication to identify the occurrence of a particular gaming venuestate.

At step 512 the AMS application can also respond to a user selection ofa profile. A profile can be a device profile or master profile invokedby selecting GUI button 156 or 158, each of which can identify theassociation of gaming actions with input functions of one or moreaccessories. If a profile selection is detected in step 512, the AMSapplication can retrieve in step 514 macro(s) and/or prior associationsdefined by the profile. The actions and/or macros defined in the profilecan also be presented in step 516 by the AMS application in the actionscolumn 130 of the GUI 101 to modify existing profile associations orcreate new associations.

In step 518, the AMS application can also respond to a user selection tocreate a macro. A macro in the present context can mean any actionablecommand which can be recorded by the AMS application. An actionablecommand can represent a sequence of stimuli generated by manipulatinginput functions of an accessory, a combination of actions in the Actionsection 130, an identification of a software application to be initiatedby the OS of the computing device 206, or any other recordable stimulusto initiate, control or manipulate software applications. For instance,a macro can represent a user entering the identity of a softwareapplication (e.g., instant messaging tool) to be initiated by the OSupon the AMS application detecting a speech command using speechrecognition technology.

A macro can also represent recordable speech delivered by a microphonesingly or in combination with a headset for detection by anothersoftware application through speech recognition or for delivery of therecorded speech to other parties. In yet another embodiment a macro canrepresent recordable navigation of an accessory such as a joystick ofthe gaming controller 115, recordable selections of buttons of thegaming controller 115, and so on. Macros can also be combinations of theabove illustrations with selected actions from the Actions 130 menu.Macros can be created from the GUI 101 by selecting a “Record Macro”button 148. The macro can be given a name and category in user-definedfields 140 and 142.

Upon selecting the Record Macro button 148, a macro can be generated byselection of input functions on an accessory (e.g., Ctrl A, speech,navigation knob movements of the gaming controller 115, etc.) and/or bymanual entry in field 144 (e.g., typing the name and location of asoftware application to be initiated by an OS, such as an instantmessaging application, keyboard entries such as Ctrl A, etc.). Once themacro is created, it can be tested by selecting button 150 which canrepeat the sequence specified in field 144. The clone button 152 can beselected to replicate the macro sequence if desired. Fields 152 can alsopresent timing characteristics of the stimulation sequence in the macrowith the ability to modify and thereby customize the timing of one ormore stimulations in the stimulation sequence. Once the macro has beenfully defined, selection of button 154 records the macro in step 520.The recording step can be combined with a step for adding the macro tothe associable items Actions column 130, thereby providing the user themeans to associate the macro with input functions of the accessories(e.g., one or more keys of the keyboard 108, buttons of the gamingcontroller 115, etc.).

In step 522, the AMS application can respond to drag and dropassociations of actions with input functions of the keyboard 108 or thegaming controller 115. Associations can also be made based on the two orthree dimensional movements of the gaming controller 115. If user inputindicates that a user is performing an association, the AMS applicationcan proceed to step 524 where it can determine if a profile has beenidentified in step 512 to record the association(s) detected. If aprofile has been identified, the associations are recorded/stored in theprofile in step 526. If a profile has not been identified in step 512,the AMS application can create a profile in step 528 for recording thedetected associations. In the same step, the user can name the newlycreated profile as desired. The newly created profile can also beassociated with one or more gaming software applications in step 530 forfuture reference. The AMS application can also record in a profile instep 526 associations based on gaming venue states. In this embodimentthe same stimuli generated by the gaming controller 115 can result indifferent substitutions based on the gaming venue state detected by theAMS application.

Referring back to step 526, once the associations have been recorded ina profile, the AMS application can determine in step 532 which of theaccessories shown illustratively in FIGS. 1-3 are programmable andavailable for programming. If the AMS application detects that anaccessory (e.g., keyboard 108, gaming controller 115) is communicativelycoupled to the computing device 206 and determines that the accessory iscapable of performing stimulus substitutions locally, the AMSapplication can proceed to step 534 of FIG. 5 where it submits theprofile and its contents for storage in the accessory (e.g., the gamingcontroller 115 in FIGS. 2-3 ). Once the accessory (e.g., the gamingcontroller 115) is programmed with the profile, the accessory canperform stimuli substitutions according to the associations recorded bythe AMS application in the profile. Alternatively, the AMS applicationcan store the profile in the computing device 206 of FIGS. 2-3 andperform substitutions of stimuli supplied by the gaming controller 115according to associations recorded in the profile by the AMSapplication.

The GUI 101 of FIG. 1 presented by the AMS application can have otherfunctions. For example, the GUI 101 can present a layout of theaccessory (button 122), or can present how the accessory is illuminatedwhen associations between input functions and actions are made (button124), and configuration options for the accessory (button 126). The AMSapplication can adapt the GUI 101 to present more than one functionalGUI page. For instance, by selecting button 102, the AMS application canadapt the GUI 101 to present a means to create macros and associateactions to accessory input functions as depicted in FIG. 1 . Selectingbutton 104 can cause the AMS application to adapt the GUI 101 to presentstatistics from stimulation information and/or gaming action resultscaptured by the AMS application as described in the subject disclosure.Selecting button 106 can also cause the AMS application to adapt the GUI101 to present promotional offers and software updates.

The steps of method 500 in whole or in part can be repeated until adesirable pattern is achieved of associations between stimulus signalsgenerated by accessories and substitute stimuli. It would be apparent toan artisan with ordinary skill in the art that there can be numerousother approaches to accomplish the embodiments described by method 500or variants thereof. These undisclosed approaches are contemplated bythe subject disclosure.

FIG. 6 depicts a method 600 for illustrating additional operations ofthe AMS application. In the configurations of FIGS. 2-3 , the AMSapplication can be operating in whole or in part from the gamingcontroller 115, a gaming console 206, a computer 206, or a remote server(not shown). For illustration purposes, it is assumed the AMSapplication operates from the gaming console 206. Method 600 can beginwith the AMS application establishing communications in steps 602 and604 between the gaming console 206 and a gaming accessory such as thegaming controller 115, and a headset 114 such as shown in FIG. 1 . Thesesteps can represent for example a user starting the AMS application fromthe gaming console 206 and/or the user inserting at a USB port of thegaming console 206 a connector of a USB cable tethered to the gamingcontroller 115, which invokes the AMS application. In step 606, thegaming controller 115 and/or headset 114 can in turn provide the AMSapplication one or more accessory ID's, or the user can provide by wayof a keyboard or the gaming controller 115 user identification. With theaccessory ID's, or user input the AMS application can identify in step608 a user account associated with the gaming controller 115 and/orheadset 114. In step 610, the AMS application can retrieve one or moreprofiles associated with the user account.

In step 612, the user can be presented by way of a display coupled tothe gaming console 206 profiles available to the user to choose from. Ifthe user makes a selection, the AMS application proceeds to step 614where it retrieves from the selected profiles the association(s) storedtherein. If a selection is not made, the AMS application can proceed tostep 616 where it can determine whether a software gaming application(e.g., video game) is operating from the gaming console 206 or whetherthe gaming console 206 is communicating with the software gamingapplication by way of a remote system communicatively coupled to thegaming console 206 (e.g., on-line gaming server(s) presenting, forexample, World of Warcraft™). If a gaming software application isdetected, the AMS application proceeds to step 617 where it retrieves aprofile that matches the gaming application detected and theassociation(s) contained in the profile. As noted earlier,association(s) can represent accessory stimulations, navigation andspeech, the invocation of other software applications, macros or othersuitable associations that result in substitute stimulations. Theaccessory stimulations can be stimulations that are generated by thegaming controller 115, as well as stimulations from other accessories(e.g., headset 114), or combinations thereof.

Once a profile and its contents have been retrieved in either of steps614 or step 617, the AMS application can proceed to step 719 of FIG. 7where it monitors for a change in a gaming venue state based on thepresentations made by the gaming application, or API messages suppliedby the gaming application. At the start of a game, for example, thegaming venue state can be determined immediately depending on the gamingoptions chosen by the gamer. The AMS application can determine thegaming venue state by tracking the gaming options chosen by a gamer,receiving an API instruction from the gaming application, or byperforming image processing on the video presentation generated by thegaming application. For example, the AMS application can detect that thegamer has directed an avatar to enter a tank. The AMS application canretrieve in step 719 associations for the gaming controller 115 forcontrolling the tank.

The AMS application can process movements of the gaming controller 115forwards, backwards, or sideways in two or three dimensions to controlthe tanks movement. Similarly, rotating the gaming controller 115 ortilting the gaming controller 115 forward can cause an accelerometer,gyro or magnetometer of the gaming controller 115 to providenavigational data to the AMS application which can be substituted withan action to cause the tank to turn and/or move forward. The profileretrieved by the AMS application can indicate that the greater theforward tilt of the gaming controller 115, the greater the speed of thetank should be moving forward. Similarly, a rear tilt can generatenavigation data that is substituted with a reverse motion and/ordeceleration of the forward motion to stop or slow down the tank. Athree dimensional lift of the mouse can cause the tank to steeraccording to the three dimensional navigation data provided by thegaming controller 115. For example, navigation data associated with acombination of a forward tilt and right bank of the gaming controller115 can be substituted by the AMS application to cause an increase inforward speed of the tank with a turn to the right determined by the AMSapplication according to a degree of banking of the gaming controller115 to the right. In the above embodiment, the three dimensionalnavigation data allows a gamer to control any directional vector of thetank including speed, direction, acceleration and deceleration.

In another illustration, the AMS application can detect a new gamingvenue state as a result of the gamer directing the avatar to leave thetank and travel on foot. Once again the AMS application retrieves instep 719 associations related to the gaming venue state. In thisembodiment, selection of buttons of the gaming controller 115 can beassociated by the AMS application with weaponry selection, firing,reloading and so on. The movement of the gaming controller 115 in two orthree dimensions can control the direction of the avatar and/orselection or use of weaponry. Once the gaming venue state is detected instep 719, the AMS application retrieves the associations related to thevenue state, and can perform substitutions of stimuli generated by thegaming controller 115, and/or speech commands received by microphone ofthe headset 114.

In one embodiment, the AMS application can be configured in step 719 toretrieve a profile that provides substitute stimuli for replacingcertain stimuli generated by accessories. The associations recorded inthe profile can be venue independent. In another embodiment, the AMSapplication can retrieve a combination of profiles, where one or moreprofiles provide substitute stimuli that are venue dependent and one ormore other profiles provide substitute stimuli that are venueindependent.

The AMS application can monitor in step 720 stimulations generated bythe accessories coupled to the gaming console 206. The stimulations canbe generated by the gamer by manipulating the gaming controller 115,and/or by generating speech commands detected by a microphone of theheadset 114. If a stimulation is detected at step 720, the AMSapplication can determine in step 722 whether to forward the detectedstimulation(s) to an Operating System (OS) of the gaming console 206 orthe gaming application directly without substitutions. Thisdetermination can be made by comparing the detected stimulation(s) tocorresponding associations in one or more profiles retrieved by the AMSapplication. If the detected stimulation(s) match the associations, thenthe AMS application proceeds to step 740 where it retrieves substitutestimulation(s) in the profile(s). In step 742, the AMS application cansubstitute the detected stimulation(s) with the substitute stimulationsin the profile(s).

In one embodiment, the AMS application can track in step 744 thesubstitute stimulations by updating the stimulations with a uniqueidentifier such as a globally unique identifier (GUID). In thisembodiment, the AMS application can also add a time stamp to eachsubstitute stimulation to track when the substitution was performed. Inanother embodiment, the AMS application can track each substitutestimulation according to its order of submission to the gamingapplication. For instance, sequence numbers can be generated for thesubstitute stimulations to track the order in which they were submittedto the gaming application. In this embodiment, the substitutestimulations do not need to be updated with sequence numbers oridentifiers so long as the order of gaming action results submitted bythe gaming application to the AMS application remain in the same orderas the substitute stimulations were originally submitted.

For example, if a first stimulation sent to the gaming application bythe AMS application is a command to shoot, and a second stimulation sentto the gaming application is a command to shoot again, then so long asthe gaming application provides a first a game action result for thefirst shot, followed by a game action result for the second shot, thenthe substitute stimulations will not require updating with sequencenumbers since the game action results are reported in the order that thestimulations were sent. If on the other hand, the game action resultscan be submitted out of order, then updating the stimulations withsequence numbers or another suitable identifier would be required toenable the AMS application to properly track and correlate stimulationsand corresponding gaming action results.

Referring back to step 722, if the detected stimulation(s) do not matchan association in the profile(s), then the AMS application proceeds toone of steps 744 or 746 in order to track the stimulations of theaccessory as described above. In another embodiment, tracking oforiginal stimulations or substitute stimulations can be bypassed byskipping steps 744 or 746 and proceeding to step 770 of FIG. 7B.

Once the stimulations received in step 720 have been substituted withother stimulations at step 742 responsive to a detected association, ormaintained unchanged responsive to detecting no association withsubstitute stimuli, and (optionally) the AMS application has chosen aproper tracking methodology for correlating gaming action results withstimulations, the AMS application can proceed to step 771 of FIG. 7B.

Referring now to FIG. 7B, a method 770 for providing a virtualization ofan accessory device in a VR OS. Many VR applications rely on the use ofVR goggles to provide the VR experience to users. In these situations,it is impossible for the user to see the accessory device or their ownbody (e.g., their hands). Therefore, all interactions with the accessorydevice, including the positioning of the accessory device and therelative positioning of their body with respect to the accessory devicemust be performed “blind” without the benefit of quick glances at theaccessory device or even the use of peripheral vision to aid inuser-device interactions.

In one or more embodiments, the method 770 can determine interactionsbetween a user, typically a user's hands, and an accessory device, suchas a keyboard device 108. These hand-device interactions can createstimulations that can be evaluated by the AMS, which can pass thestimulations on and/or replace/augment the stimulations with substitutestimulations (as described above). For example, if a user's handinteracts with a keyboard device 108 to press a down arrow key, the AMScan detect this stimulation, determine if a substitution is in order,and then pass this stimulation and/or substitute stimulation to the VROS. The features and various embodiments described for the method 770can be performed by the AMS, the accessory device, the VR OS or anycombination thereof.

At step 771, light can be projected into an operational space of theaccessory device. In one embodiment, the light can be infrared light orlight of some other particular frequency. For example, if light in theinfrared spectrum is projected over an operational area of a keyboarddevice 108, then interactions between the user's hand(s) and thekeyboard device 108 can be captured or detected by capturing images inthe infrared spectrum. In another embodiment, the projected light can bein a different spectrum, where this light is captured at a camera 428that filters out other light components (i.e., light in other parts ofthe spectrum).

At step 772, light from the operational space can be received at thecamera 428. In one embodiment, the captured light can be reflected fromthe hand(s) of the user. In another embodiment, the captured light canbe emitted from the hand(s) of the user. For example, heat radiatingfrom the user's hands can cause infrared light to be emitted from theuser's hands. The camera 428 can collect this light. In one embodiment,no light is projected into the operational space. Rather, the camera 428simply captures ambient light from the operational space. The capturedlight can be infrared light, such as light emitted from the user'shand(s). The captured light can include light in the visible spectrum.

At step 774, the user's hand image can be extracted from the light thatis received. In one embodiment, a pattern recognition technique can beapplied to the captured light. For example, where the expected image isa hand or a pair of hands interacting with a keyboard device 108, thecaptured light can be analyzed to detected image components that areconsisted with a hand or a pair of hands. At step 776, hand featureinformation can be determined from the hand image. In one embodiment,where the captured light is infrared light, a pattern recognitiontechnique can analyze the captured hand image for areas of differinglight intensity. For example, fingers or fingertips may give off greaterlevels of heat than other parts of the user's hand. Therefore, fingersand/or fingertips can emit greater intensity of infrared light, whichcan be used to determine particular finger locations. In one embodiment,the pattern recognition can use training techniques, such as capturinglight while the user is instructed to place her hand(s) in theoperational area and/or types a known pattern into the keyboard device108. The hand feature information can provide real time hand positions,locations, shapes, and movements. In one embodiment, the keyboard device108 can be constructed of a material that absorbs light in a particularpart of the spectrum. For example a keyboard that absorbs infrared lightcan serve as an excellent “black” background for the user's hand(s),which can emit and/or reflect infrared light. In this scenario thecamera can capture the infrared light form the hand(s) while theabsorbing background keyboard device 108 can improve the signal-to-noiseratio (SNR) of the light signal for the hand(s). In one embodiment, animage of the user's hand(s) can be captured and processed in the visiblelight spectrum.

At step 778, location information for the accessory device can beextracted from the light that is received from the operational space. Inone embodiment, the accessory device can emit light or can reflect lightthat projected onto it. For example, a keyboard device 108 can have aseries of light emitting/reflecting markings in a known pattern. Themarkings can be used to determine a location and an orientation of thekeyboard device 108 with respect to the camera 428. In one embodiment,the relative location of the camera 423 and the keyboard device 108 canbe fixed and known. For example, the camera 423 can be fixed in locationwith the respect to the keyboard device 108, such as would be the caseif the camera 423 is mounted onto the keyboard device 108. In this case,it may not be necessary to derive location information for the accessorydevice from the captured image data.

At step 780, the hand feature information and the location informationcan be mapped to generate hand-device interaction information for theoperational space. In one embodiment, the location information for theaccessory device can provide a known “context” for the determined handfeature information. As each location and/or movement of the handfeatures plays out over the accessory device context, the hand-deviceinteractions can indicate a sequence of key strokes. If the accessorydevice location information is known and fixed with respect to thecamera 423, such as in the case of a fixed camera attached to thekeyboard, then the hand feature information can simply be mapped onto aknown template of the keyboard device 108.

In one or more embodiments, stimulation and/or substitute stimulationinformation for the accessory device can be accessed from the AMS. Thatis, the user input stimulation that the AMS has captured from the user'sinteractions with the accessory device can be accessed. At step 784, thestimulation and/or substitute stimulation information can be compared tothe hand-device interaction information. For example, where AMS hasreceived a user input of “up arrow” from a keyboard device 108, thisstimulation can be compared to a concurrently determined hand-deviceinteraction that is determined, based on pattern recognition of the handimage and location information for the keyboard device, to alsocorrespond to an “up arrow” input. In this case, the hand-deviceinteraction information and the AMS stimulation are in agreement, atstep 786. Alternatively, the hand-device interaction “keystroke” may notmatch the “keystroke” that is determined by AMS. For example, AMS maydetect a stimulation of “up arrow,” while the hand image captured at thecamera and mapped to the keyboard device location information maygenerate a hand-device interaction that indicates a user strike of the“page down” key. In this case, the hand-device interaction informationand the AMS “keystroke” stimulation are not in agreement, at step 786.In this case, the hand-device interaction information can be correctedso that it agrees with the AMS stimulation and/or substitutestimulation. In one embodiment, the AMS stimulation and/or substitutestimulation is preferred over the image-determined “keystroke.” Forpurposes of presentation at the VR application, the corrected“keystroke” for the hand-device interaction may be preferred.

At step 794, the hand-device interaction information can be transmittedto the VR OS. In one embodiment, AMS can submit the stimulation and/orsubstitute stimulation to the VR OS concurrently with the hand-deviceinteraction information. The VR OS can provide the hand-deviceinteraction information and the AMS stimulation information to a VRapplication. The VR application can translate the hand-deviceinformation into the VR domain and, in turn, present the accessorydevice and/or the user's hand(s) to the user in the context of the VRdisplay. For example, the VR application can present the accessorydevice and/or the user's hand(s) at a VR goggle device. In oneembodiment, an actual image of the hand(s) and accessory device can besent to the VR OS. The VR OS can then virtualize the accessory deviceand operator hand(s) based on the actual image from the camera 423.

Once the AMS application at step 748 supplies to a Virtual RealityOperating System (OS) or a Virtual Reality Application of the computingdevice 206 a gaming action (i.e., one or more stimulations) and thehand-device interaction information from the methods of FIGS. 7A and 7B,the AMS application can proceed to step 734. The gaming action suppliedto the Virtual Reality OS at step 748 can be the unadulterated“original” gaming action of step 770, or an alternative gaming actiongenerated by steps 794 or 796. At step 734, the Virtual Reality OSdetermines whether to invoke in step 736 a software applicationidentified in the stimulation(s) (e.g., gamer says “turn on team chat”,which invokes a chat application), whether to forward the receivedstimulation(s) to the gaming software application in step 738, orcombinations thereof.

Contemporaneous to the embodiments described above, the AMS applicationcan monitor in step 750 for game action results supplied by the gamingapplication via API messages previously described. For instance, supposethe stimulation sent to the gaming application in step 738 is a commandto shoot a pistol. The gaming application can determine that the shotfired resulted in a miss of a target or a hit. The gaming applicationcan respond with a message which is submitted by way of the API to theAMS application that indicates the shot fired resulted in a miss or ahit. If IDs such as GUIDs were sent with each stimulation, the gamingapplication can submit game action results with their corresponding GUIDto enable the AMS application to correlate the gaming action resultswith stimulations having the same GUID.

For example, if the command to shoot included the ID “1234”, then thegame action result indicating a miss will include the ID “1234”,enabling the AMS application in step 752 to correlate the game actionresult to the stimulation having the same ID. If on other hand, theorder of game action results can be maintained consistent with the orderof the stimulations, then the AMS application can correlate in step 754stimulations with game action results by the order in which stimulationwere submitted and the order in which game action results are received.In step 756, the AMS application can catalogue stimulations and gameaction results. In another embodiment, the AMS application can beadapted to catalogue the stimulations in step 760. In this embodiment,step 760 can be performed as an alternative to steps 750 through 756. Inanother embodiment, step 760 can be performed in combination with steps750 through 756 in order to generate a catalogue of stimulations, and acatalogue for gaming action results correlated to the stimulations.

FIG. 8 illustrates embodiments of an accessory device 800 that can bevirtualized into a VR Operating System or VR application. In one or moreembodiments, the accessory device 800 can be a user input device, akeyboard device, a mouse device, game controller device, a joystickdevice, a device that combines keypad and control functions, and/or adevice that combines input and output functions, such as hapticfeedback. In one embodiment, the accessory device can combine computeraccessory functions, such as a keyboard, with an image capturingfunction and a Virtual Reality (VR) communication function. For example,the accessory device 800 can include a keyboard device 804, a cameradevice 816, and a VR communication device 824. In this embodiment, thecamera device 816 and the VR communication device 824 are attached tothe keyboard device 804 using a mounting arm 808. In one embodiment, theaccessory device 800 can include a light source 812 that is alsoattached to the keyboard device 800 via the mounting arm 808.

In one embodiment, the mounting arm 808 can position the camera device816 over the keyboard device 804 such that the camera device 816 cancapture images from an operational area over the keyboard device 804.The operational area can include an area where a user of the keyboarddevice 804 would place his/her hands while using the keyboard. In oneembodiment, the mounting arm 808 can position the light source 812 overthe keyboard device 804 such that the light source 812 can project lightonto an operational area of the keyboard device 804.

In one or more embodiments, the VR communication device 824 can receivedata from the camera device 816. In various embodiments, the data fromthe camera device 816 can be unprocessed, partially processed, orcompletely processed before it is provided to the VR communicationdevice 824. The VR communication device 824 can transmit data from thecamera device 816 to a VR Operating System for use in presenting avirtualized version of the keyboard and the user's hand(s) in a VRapplication.

FIGS. 9A-9B illustrate embodiments a desktop system 900 using thevirtualizing accessory device 800. In one or more embodiments, thedesktop system 900 can include a computer device 908, which may includea viewing screen and the accessory device 800. The desktop system 900can include VR goggles 904, which can provide a VR viewing environmentfor the user. The accessory device 800 can be used as an input device tothe computer device 908. The user can interact with the accessory device800 to control a VR application that is executing at the computer device908 or that is executing remotely, such as on a remote server, whilevideo, audio, and control data is streamed through the computer device908. In one or more embodiments, the camera device 816 if the accessorydevice 800 can capture video and/or still image information from anoperational area 912 of the accessory device 800. The camera device 816can capture imaging information that includes the keyboard device 804and the user's hand(s) as they interact with the keyboard device 804. Alight source 812, such as an Infrared Emitter, can be included in theaccessory device 800. The light source 812 can project light into theoperational areal 912 of the keyboard device 804. The projected lightcan be reflected by objects in the operational area 912, including thekeyboard device 804 and the hand(s) of the user.

FIG. 10 illustrates embodiments for detecting hand images and/or handfeature information using the accessory device 800. In one or moreembodiments, the accessory device 800 can detect hand images and/or handfeature information by capturing images (light) from the operationalarea 912. In various embodiments, the hand image and/or featureinformation can be captured using an image from the visible lightspectrum, or from a part of the spectrum (e.g., infrared light).

In one or more embodiments, the accessory device 800 can project lightinto the operational area 912 using the light source 812. In oneexample, an Infrared (IR) spectrum light source 812 can be used. IRlight can be invisible to the naked eye and, therefore, can provide alight source that does not distract from the user or other people in theroom. In addition, by providing light of a known wavelength (orwavelength range) to the operational area 912, the camera device 816 canemploy filtering techniques to block out light of other wavelengthswhile receiving a strong reflected light signal for processing. Inanother embodiment, the user may wear reflective gloves or finger cotson his/her hands 102. The reflective gloves or cots can be made toreflect light in the wavelength of the light source 812, while absorbingother wavelengths of light, to increase the light signal-to-noise ratio(SNR) in the wavelength of interest. The camera device 816 can receivethe light that is reflected from the hand(s) 1020 at hand features ofinterest, such as the fingertips 1040.

In one or more embodiments, the accessory device 800 can omit the lightsource 812 and simply capture images from the operational area 912 basedon ambient light and/or light that is emitted from the hand(s) 1020and/or the keyboard 804. For example, heat from the hand(s) 1020 cangive off radiation in the IR spectrum. The camera device 816 can capturethe emitted IR from the hand(s) 1020 and use this for processing handimage and hand feature information. In one embodiment, the user can weargloves that accentuate the IR emission in areas of interest on thehand(s) 1020. For example, gloves could expose fingertips 1040 whilecovering other parts of the hand(s) 1020 so that IR emission fromnon-fingertip areas are significantly reduced or eliminated. In thisway, IR emissions from the fingertip areas are accentuated in the IRimages captured at the camera device 816. In one or more embodiments,the camera device 816 can capture light that is emitted from the hand(s)1020 and/or keyboard device 804 along with light that is reflected fromthe hand(s) 1020 and/or keyboard device 804.

In one embodiment, light can be reflected or emitted from the keyboarddevice 804 at specific locations on the keyboard device 804. Forexample, positioning and/or alignment markings 1004 can be included onthe keyboard device 804, where these markings can reflect light at aspecific wavelength (e.g., IR light coming from the light source 812) oremit light at a specific wavelength (e.g., IR light coming from a lightsource internal to the keyboard device 804. Keyboard markings 1004 canprovide the camera device 816 with positive location and orientationinformation. A single marking 1004 can allow the keyboard device 804 tobe located with respect to the camera device 816, while multiplemarkings 1004 can provide rotational information. In one or moreembodiments, the location and/or orientation of the keyboard device 804can be fixed and known in relation to the camera device 816. Forexample, where the camera device 816 is attached to the keyboard device804, as with the mounting arm 808, the relative location of the cameradevice 816, with respect to the keyboard device 804, can be known andunchanging. In this case, it may not be necessary to determine thelocation and/or orientation of the keyboard device 804, and knownlocation/orientation information may be used for purposes of providingcontext to the user's hand(s).

As described above with respect to FIG. 7B, images received at thecamera device 816 can include information associated with theoperational area above the keyboard device 804. The camera device 816can transmit raw image data to the VR communication device 824 or canprocess the image data prior to sending it to the VR communicationdevice 824. In one embodiment, the camera device 816 can process rawimage data to detect hand images and/or hand feature information. In oneembodiment, the VR communication device 824 can receive the raw imagefrom the camera device 816, and the VR communication device 824 canprocess the raw image data into hand images and/or hand featureinformation. In one embodiment, the VR communication device 824 cantransmit the raw image data to the VR Operating System, which canprocess the raw image data into hand images and/or hand featureinformation.

In one or more embodiments, the hand feature information, which isprocessed from the captured image data, can include information in theform of modeling data for the hand(s) 1020 of the user. The modelingdata can include location and vectoring information for the features ofeach hand 1020 that are extracted from the hand images captured at thecamera device 816. The modeling data can be input into software handmodels to reproduce the movements of the key features—the fingers andthumbs, as well as the macro movements of the hands.

In one or more embodiments, the hand feature information and thelocation information for the keyboard device 804 can be mapped together,or combined, at the camera device 816, the VR communication device 824,or the VR Operating System, to generate hand-device interactioninformation. In this process, the physical location and movements of thehand(s) 1020 can be mapped to the physical location and orientation ofthe keyboard device 804 such that the contextual relationship betweenthe hand(s) 1020 and the keyboard device 804 are modeled. For example,the camera can capture a movement of the right ring finger thatcorresponds to pressing of a keyboard character, and, when this movementis mapped to the current location and orientation of the keyboard device804, the apparent key press is determined to be the letter “L.” Whenmapped together, hand-device interaction information is generated. Thishand-device interaction information can be used to generate a real-time,VR hand-keyboard 1100 of the hand(s) 1140 and the keyboard device 1104,as shown in FIG. 11 . Each real-time movement of the user's hand(s) 1020in the operational area 912 of keyboard device 804 is thereby captured,processed, and converted into a real-time set of hand-device interactiondata that tracks relational movements of the real hand(s) 1020 and thereal keyboard device 804 and translates these real world movements intoVR movements in real-time. As a result, when the user's real right ringfinger of his real hand(s) 1020 strikes the letter “L” on the realkeyboard device 804, then hand-device interaction information reflectsthis movement, in real-time, and a VR recreations of the hand(s) 1120and keyboard 1104 in the VR environment of FIG. 11 , will reflect thismovement.

In one or more embodiments, the hand-device interaction information canbe corrected using information from the AMS system. The correction canbe performed at the camera device 816, the VR communication device 824,and/or the VR Operating System. In one embodiment, the AMS system canprovide stimulation and/or substitution information for the keyboarddevice 804. The stimulation information can be captured by the AMS basedon actual keystrokes or user inputs into the keyboard device 804, and,therefore, the stimulation information can be an accurate reflection ofthe user interactions that are occurring in real-time. The hand-deviceinteraction information that is derived from images captured at thecamera device 816 can be compared to the AMS-based stimulationinformation. Where there are discrepancies between the AMS-basedstimulations and the image-based hand-device interaction information,the image-based hand-device interaction information can be corrected.For example, image-based hand-device interaction information mayindicate that the user is hitting a series of keystrokes that are offsetfrom the actual AMS-based stimulations by one key (e.g., the images areoffset one key to the right of the real key strokes). Rather thanrecreating this error in a VR version of the hand(s) 1140 and keyboarddevice 1104, the hand-device interaction information can be corrected to“shift” the keyboard location to the right. As a result, the VR keyboard1104 that is reproduced in the VR environment will shift to theright—with respect to the VR hands 1140—and the user will be presentedwith a VR hand-keyboard 1100 that better reflect the real-worldsituation. In one or more embodiments, the VR communication device 824can send actual images of the user hand(s) and/or keyboard device 804 tothe VR Operating System. In such a case, the VR Operating System may becapable of superimposing the actual images of the user hand(s) and/orkeyboard device 804 on images from a VR application.

FIGS. 12A-12B depict illustrative embodiments of how the VR hands 1140and VR keyboard 1120 can be overlaid onto a VR display 1220 for a VRgaming application 1200 and a VR display 1270 for a VR businessapplication 1250. In each case, the user of VR goggles can obstruct theuser's view of his/her hands and keyboard. The VR hands 1140 and VRkeyboard 1120 provide the user with real-time visibility of therelationship between hand(s) and keyboard and allow them to quicklyreference keypad locations, even from their peripheral vision, whilestaying entirely in the VR environment.

In one or more embodiments, the VR communication device 824 can includea data path to input data into the VR Operating System. One example of aVR communication device can be a Vive Tracker™, which is manufactured bythe HTC Corporation (Taiwan). One example of a VR Operation System isSteamOS™, which was developed by the Valve Corporation (United States).In one embodiment, the hand-device interaction information can beprovided to the VR communication device 824 via a direct connection tothe camera device 816. The VR communication device 824 can engage in awireless communication session with the VR Operating System, which canbe executing at the computer device 904. The VR communication device 824can translate the hand-device interaction information for the hand andkeyboard model to a format that fits the VR Operating System beforetransmitting the hand-device interaction information to the VR OperatingSystem. The VR Operating System can then provide the hand-deviceinteraction information to a VR application, using, for example, a VRAPI. The VR application can use the hand-device interaction informationto generate the VR versions of the hand(s) 1140 and the keyboard 1104for overlay onto the current window into the VR environment.

In one or more embodiments, the VR communication device 824 can receivethe AMS stimulation and/or stimulation substitution information andtransmit the stimulation and/or stimulation substitution information tothe VR Operating System using the wireless communication session. The VROperating System can provide the stimulation and/or stimulationsubstitution information to the VR application, which can use thestimulation and/or stimulation substitution information as userinput/control for the VR application.

In one or more embodiments, hand images and hand feature information canbe captured and determined using a CMOS image sensor based on a gesturerecognition analysis. In gesture recognition, reflections andabsorptions of the lighting source 812 interacting with the user'shand(s) and/or finger(s) is used to capture and compare hand gestures.These gestures can correspond to known hand forms used during the entryof data and/or entry of movement commands (e.g., left arrow, up arrow)into the keyboard device 804.

In one embodiment, the light source 812 can be an Infrared LightEmitting Diode (LED). In one embodiment, the keyboard device 804 canhave a top surface that, generally, absorbs light in the wavelength thatis projected from the light source 812. For example, the keyboard device804 can generally absorb Infrared light—with the exception of thepositional markings 1004, which can reflect Infrared light. By placingthe user's hand(s) 1020 above this generally absorbing background, mostof the Infrared light that is reflected and captured at the cameradevice 816 will be coming from the user's hand(s) 1020. The cameradevice 816 can identify the outline of the hands and use patternrecognition and/or Artificial Intelligence (AI) training to identity theposition of the finger tips 1040 with respect to the keyboard device804. This positioning can be made more certain with it is combined withthe keypad presses that are tracked by the AMS system. By combining thetwo—Infrared images and key strokes—an accurate Three-dimensional (3D)model of the hand/finger action with respect to the keyboard device 804can be determined.

In one or more embodiments, the real-time, hand-device interactioninformation can be injected into the VR Operating System via the VRcommunication device 824. The VR Operating System can then provide the3D model information to overlay a Virtual keyboard and virtual handsonto the Virtual application.

FIGS. 13-14 illustrate embodiments of a system with a correspondingcommunication flow diagram for correlating stimulations and gamingaction results. In this illustration a user clicks the left button 119of the gaming controller 115. The gaming controller 115 can includefirmware (or circuitry), which creates an event as depicted by event 2in FIG. 13 . The button depression and the event creation are depictedin FIG. 14 as steps 1402 and 1404. In step 1404, the firmware of thegaming controller 115 can, for example, generate an event type “leftbutton #3”, and a unique GUID with a time stamp which is submitted tothe AMS application. Referring back to FIG. 13 , the AMS applicationcatalogues event 3, and if a substitute stimulation has been predefined,remaps the event according to the substitution. The remapped event isthen transmitted to the gaming application at event 4. Event 3 of FIG. 8is depicted as step 1406 in FIG. 14 . In this illustration, the AMSapplication substitutes the left button #3 depression stimulus with a“keyboard ‘F’” depression which can be interpreted by the gamingapplication as a fire command. The AMS application in this illustrationcontinues to use the same GUID, but substitutes the time stamp foranother time stamp to identify when the substitution took place.

Referring back to event 4, the gaming application processes the eventand sends back at event 5 a game action result to the AMS applicationwhich is processed by the AMS application at event 6. The AMSapplication then submits the results to the accessory at event 7. Events4 and 5 are depicted as step 1408 in FIG. 14 . In this step, the gamingapplication processes “F” as an action to fire the gamer's gun, and thendetermines from the action the result from logistical gaming resultsgenerated by the gaming application. In the present illustration, theaction of firing resulted in a hit. The gaming application submits tothe AMS application the result type “Hit” along with a new time stamp,while utilizing the same GUID for tracking purposes. At step 1410, theAMS application correlates the stimulation “left button #3 (and/or thesubstitute stimulation keyboard “F”) to the game result “Hit” andcatalogues them in memory. The AMS application then submits to theaccessory (e.g., gaming controller 115) in step 1410 the game actionresults “Hit” with the same GUID, and a new time stamp indicating whenthe result was received. Upon receiving the message from the AMSapplication, the accessory in step 1412 processes the “Hit” by assertinga red LED on the accessory (e.g., left button 119 illuminates in red orother LED of the gaming controller 115 illuminates in red) to indicate ahit. Other notification notices can be used such as another color forthe LED to indicate misses, a specific sound for a hit, or kill, avibration or other suitable technique for notifying the gamer of thegame action result.

Upon reviewing the aforementioned embodiments, it would be evident to anartisan with ordinary skill in the art that the embodiments of thesubject disclosure can be modified, reduced, or enhanced withoutdeparting from the scope of the claims described below. For example, theAMS application can be executed from an accessory 115 or computingdevice 206 to perform the embodiments described in the subjectdisclosure. The AMS application can also be operated from a remoteserver (“cloud services”). In yet another embodiment, functions of theAMS application can be distributed between devices. In yet anotherembodiment, the AMS application can be configured to track theperformance of a gamer and adapt a threshold as the gamer improves ordeclines in performance.

For instance, as a gamer's performance improves with a particular gamingaction, the threshold associated with the gaming action can be adaptedto be less sensitive in detecting an over usage state. Similarly, thesensitivity of the threshold can be increased to promptly identify anover usage state of a gaming action if the gamer's performance declinesas a result of an over usage of the gaming action. Additionally, the AMSapplication can be adapted to add gaming actions to an exclusion tablewhen the gamer's performance substantially improves as a result of usingthe gaming action being excluded. The exclusion table can also bechanged by the AMS application by removing a gaming action from theexclusion table responsive to its excessive use causing a decline in agamer's performance.

Other embodiments can be applied to the subject disclosure.

It should be understood that devices described in the exemplaryembodiments can be in communication with each other via various wirelessand/or wired methodologies. The methodologies can be links that aredescribed as coupled, connected and so forth, which can includeunidirectional and/or bidirectional communication over wireless pathsand/or wired paths that utilize one or more of various protocols ormethodologies, where the coupling and/or connection can be direct (e.g.,no intervening processing device) and/or indirect (e.g., an intermediaryprocessing device such as a router).

FIG. 15 depicts an exemplary diagrammatic representation of a machine inthe form of a computer system 1500 within which a set of instructions,when executed, may cause the machine to perform any one or more of themethods described above. One or more instances of the machine canoperate, for example, as an accessory, computing device or combinationsthereof. In some embodiments, the machine may be connected (e.g., usinga network 1526) to other machines. In a networked deployment, themachine may operate in the capacity of a server or a client user machinein a server-client user network environment, or as a peer machine in apeer-to-peer (or distributed) network environment.

The machine may comprise a server computer, a client user computer, apersonal computer (PC), a tablet, a smart phone, a laptop computer, adesktop computer, a control system, a network router, switch or bridge,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. It will beunderstood that a communication device of the subject disclosureincludes broadly any electronic device that provides voice, video ordata communication. Further, while a single machine is illustrated, theterm “machine” shall also be taken to include any collection of machinesthat individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methods discussed herein.

The computer system 1500 may include a processor (or controller) 1502(e.g., a central processing unit (CPU)), a graphics processing unit(GPU, or both), a main memory 1504 and a static memory 1506, whichcommunicate with each other via a bus 1508. The computer system 1500 mayfurther include a display unit 1510 (e.g., a liquid crystal display(LCD), a flat panel, or a solid state display). The computer system 1500may include an input device 1512 (e.g., a keyboard), a cursor controldevice 1514 (e.g., a mouse), a disk drive unit 1516, a signal generationdevice 1518 (e.g., a speaker or remote control) and a network interfacedevice 1520. In distributed environments, the embodiments described inthe subject disclosure can be adapted to utilize multiple display units1510 controlled by two or more computer systems 1500. In thisconfiguration, presentations described by the subject disclosure may inpart be shown in a first of the display units 1510, while the remainingportion is presented in a second of the display units 1510.

The disk drive unit 1516 may include a tangible computer-readablestorage medium 1522 on which is stored one or more sets of instructions(e.g., software 1524) embodying any one or more of the methods orfunctions described herein, including those methods illustrated above.The instructions 1524 may also reside, completely or at least partially,within the main memory 1504, the static memory 1506, and/or within theprocessor 1502 during execution thereof by the computer system 1500. Themain memory 1504 and the processor 1502 also may constitute tangiblecomputer-readable storage media.

Dedicated hardware implementations including, but not limited to,application specific integrated circuits, programmable logic arrays andother hardware devices can likewise be constructed to implement themethods described herein. Application specific integrated circuits andprogrammable logic array can use downloadable instructions for executingstate machines and/or circuit configurations to implement embodiments ofthe subject disclosure. Applications that may include the apparatus andsystems of various embodiments broadly include a variety of electronicand computer systems. Some embodiments implement functions in two ormore specific interconnected hardware modules or devices with relatedcontrol and data signals communicated between and through the modules,or as portions of an application-specific integrated circuit. Thus, theexample system is applicable to software, firmware, and hardwareimplementations.

In accordance with various embodiments of the subject disclosure, theoperations or methods described herein are intended for operation assoftware programs or instructions running on or executed by a computerprocessor or other computing device, and which may include other formsof instructions manifested as a state machine implemented with logiccomponents in an application specific integrated circuit or fieldprogrammable gate array. Furthermore, software implementations (e.g.,software programs, instructions, etc.) including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods described herein. It is furthernoted that a computing device such as a processor, a controller, a statemachine or other suitable device for executing instructions to performoperations or methods may perform such operations directly or indirectlyby way of one or more intermediate devices directed by the computingdevice.

While the tangible computer-readable storage medium 1522 is shown in anexample embodiment to be a single medium, the term “tangiblecomputer-readable storage medium” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “tangible computer-readable storage medium” shallalso be taken to include any non-transitory medium that is capable ofstoring or encoding a set of instructions for execution by the machineand that cause the machine to perform any one or more of the methods ofthe subject disclosure. The term “non-transitory” as in a non-transitorycomputer-readable storage includes without limitation memories, drives,devices and anything tangible but not a signal per se.

The term “tangible computer-readable storage medium” shall accordinglybe taken to include, but not be limited to: solid-state memories such asa memory card or other package that houses one or more read-only(non-volatile) memories, random access memories, or other re-writable(volatile) memories, a magneto-optical or optical medium such as a diskor tape, or other tangible media which can be used to store information.Accordingly, the disclosure is considered to include any one or more ofa tangible computer-readable storage medium, as listed herein andincluding art-recognized equivalents and successor media, in which thesoftware implementations herein are stored.

Although the present specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosure is not limited to such standards andprotocols. Each of the standards for Internet and other packet switchednetwork transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP) representexamples of the state of the art. Such standards are from time-to-timesuperseded by faster or more efficient equivalents having essentiallythe same functions. Wireless standards for device detection (e.g.,RFID), short-range communications (e.g., Bluetooth®, WiFi, Zigbee®), andlong-range communications (e.g., WiMAX, GSM, CDMA, LTE) can be used bycomputer system 1500.

The illustrations of embodiments described herein are intended toprovide a general understanding of the structure of various embodiments,and they are not intended to serve as a complete description of all theelements and features of apparatus and systems that might make use ofthe structures described herein. Many other embodiments will be apparentto those of skill in the art upon reviewing the above description. Theexemplary embodiments can include combinations of features and/or stepsfrom multiple embodiments. Other embodiments may be utilized and derivedtherefrom, such that structural and logical substitutions and changesmay be made without departing from the scope of this disclosure. Figuresare also merely representational and may not be drawn to scale. Certainproportions thereof may be exaggerated, while others may be minimized.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

Although specific embodiments have been illustrated and describedherein, it should be appreciated that any arrangement calculated toachieve the same purpose may be substituted for the specific embodimentsshown. This disclosure is intended to cover any and all adaptations orvariations of various embodiments. Combinations of the aboveembodiments, and other embodiments not specifically described herein,can be used in the subject disclosure. In one or more embodiments,features that are positively recited can also be excluded from theembodiment with or without replacement by another component or step. Thesteps or functions described with respect to the exemplary processes ormethods can be performed in any order. The steps or functions describedwith respect to the exemplary processes or methods can be performedalone or in combination with other steps or functions (from otherembodiments or from other steps that have not been described).

Less than all of the steps or functions described with respect to theexemplary processes or methods can also be performed in one or more ofthe exemplary embodiments. Further, the use of numerical terms todescribe a device, component, step or function, such as first, second,third, and so forth, is not intended to describe an order or functionunless expressly stated so. The use of the terms first, second, thirdand so forth, is generally to distinguish between devices, components,steps or functions unless expressly stated otherwise. Additionally, oneor more devices or components described with respect to the exemplaryembodiments can facilitate one or more functions, where the facilitating(e.g., facilitating access or facilitating establishing a connection)can include less than every step needed to perform the function or caninclude all of the steps needed to perform the function.

In one or more embodiments, a processor (which can include a controlleror circuit) has been described that performs various functions. Itshould be understood that the processor can be multiple processors,which can include distributed processors or parallel processors in asingle machine or multiple machines. The processor can be used insupporting a virtual processing environment. The virtual processingenvironment may support one or more virtual machines representingcomputers, servers, or other computing devices. In such virtualmachines, components such as microprocessors and storage devices may bevirtualized or logically represented. The processor can include a statemachine, application specific integrated circuit, and/or programmablegate array including a Field PGA. In one or more embodiments, when aprocessor executes instructions to perform “operations”, this caninclude the processor performing the operations directly and/orfacilitating, directing, or cooperating with another device or componentto perform the operations.

The Abstract of the Disclosure is provided with the understanding thatit will not be used to interpret or limit the scope or meaning of theclaims. In addition, in the foregoing Detailed Description, it can beseen that various features are grouped together in a single embodimentfor the purpose of streamlining the disclosure. This method ofdisclosure is not to be interpreted as reflecting an intention that theclaimed embodiments require more features than are expressly recited ineach claim. Rather, as the following claims reflect, inventive subjectmatter lies in less than all features of a single disclosed embodiment.Thus the following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separately claimedsubject matter.

What is claimed is:
 1. A method, comprising: receiving, by a processingsystem including a processor, actual stimulation data generated by aphysical keyboard device responsive to a user interaction with thephysical keyboard device, wherein the actual stimulation datacorresponds to an actual stimulation operation; identifying, by theprocessing system, a first location of a virtual finger and a secondlocation of a virtual keyboard device; identifying, by the processingsystem, a virtual stimulation operation according to the first locationof the virtual finger and the second location of the virtual keyboarddevice; comparing, by the processing system, the actual stimulationoperation and the virtual stimulation operation to detect a difference;and modifying, by the processing system, the first location of thevirtual finger, to obtain a modified virtual finger location, responsiveto detecting the difference, wherein the modified virtual fingerlocation corresponds to an updated virtual stimulation that equates tothe actual stimulation operation.
 2. The method of claim 1, furthercomprising: transmitting, by the processing system, the modified virtualfinger location to a virtual reality system, wherein the modifiedvirtual finger location is provided to a virtual reality application bythe virtual reality system, wherein the virtual reality applicationsuperimposes a first image of the first location of the virtual fingerand a second image of the virtual keyboard device onto a virtual realitydisplay of the virtual reality application according to the modifiedvirtual finger location, and wherein an input based on the actualstimulation data generated by the physical keyboard device is providedto the virtual reality system, the input being preferred over thevirtual stimulation operation.
 3. The method of claim 2, furthercomprising: translating, by the processing system, the virtualstimulation operation into a format compatible with the virtual realitysystem prior to the transmitting of the modified virtual finger locationto the virtual reality system, wherein the virtual stimulation operationis converted to the format compatible with the virtual reality system bya conversion unit attached to the physical keyboard device.
 4. Themethod of claim 1, wherein a mapping of the first location of thevirtual finger and the second location of the virtual keyboard device isobtained from a virtual reality system.
 5. The method of claim 1,further comprising: providing, by the processing system, the actualstimulation data generated by the physical keyboard device to a virtualreality system.
 6. The method of claim 5, wherein the actual stimulationdata is provided to a virtual reality application by way of the virtualreality system.
 7. The method of claim 6, wherein the virtual realityapplication comprises a virtual reality gaming application.
 8. Themethod of claim 4, wherein the virtual reality system comprises avirtual reality operating system.
 9. The method of claim 1, furthercomprising: recording, by the processing system, an association betweenfirst stimulation data that can be generated by the physical keyboarddevice and a first substitute keyboard stimulation; receiving, by theprocessing system, the actual stimulation data from the physicalkeyboard device; detecting, by the processing system, a match betweenthe actual stimulation data that is received from the physical keyboarddevice and the association between the first stimulation data and thefirst substitute keyboard stimulation; retrieving, by the processingsystem, the first substitute keyboard stimulation responsive todetecting the match; and replacing, by the processing system, the actualstimulation data that is received from the physical keyboard device withthe first substitute keyboard stimulation to generate actual substitutestimulation data generated by the physical keyboard device prior totransmission of the actual substitute stimulation data to a virtualreality system.
 10. A non-transitory, machine-readable storage medium,comprising instructions, wherein responsive to executing theinstructions, a processing system including a processor performsoperations comprising: receiving physical stimulation data generated bya physical accessory device responsive to a user interaction with thephysical accessory device, wherein the physical stimulation datacorresponds to an actual stimulation operation; identifying a virtualstimulation operation of a virtual accessory device according to avirtual hand-device interaction of between a virtual hand and thevirtual accessory device; comparing the physical stimulation data andthe virtual stimulation operation to detect a difference; and modifyingthe virtual hand-device interaction to generate modified virtualhand-device interaction responsive to detecting the difference, whereinthe modified virtual hand-device interaction corrects the virtualhand-device interaction so that the virtual hand-device interactionindicated by an apparent location of the virtual hand and the virtualaccessory device agrees with the user interaction with the physicalaccessory device.
 11. The non-transitory, machine-readable storagemedium of claim 10, wherein the operations further comprise: providingthe modified virtual hand-device interaction to a virtual realitysystem, wherein the modified virtual hand-device interaction is providedby the virtual reality system to a virtual reality application, andwherein the virtual reality application superimposes the virtual handand the virtual accessory device onto a virtual reality display of thevirtual reality application according to the modified virtualhand-device interaction.
 12. The non-transitory, machine-readablestorage medium of claim 11, wherein the virtual reality applicationcomprises a virtual reality gaming application.
 13. The non-transitory,machine-readable storage medium of claim 10, wherein the physicalstimulation data is transmitted to a virtual reality system.
 14. Thenon-transitory, machine-readable storage medium of claim 13, wherein thevirtual reality system comprises a virtual reality operating system. 15.The non-transitory, machine-readable storage medium of claim 10, whereinthe operations further comprise: recording an association between firststimulation data that can be generated by the physical accessory deviceand a first substitute accessory stimulation; receiving the physicalstimulation data from the physical accessory device; detecting a matchbetween the physical stimulation data that is received from the physicalaccessory device and the association between the first stimulation dataand the first substitute accessory stimulation; retrieving the firstsubstitute accessory stimulation responsive to detecting the match; andreplacing the physical stimulation data that is received from thephysical accessory device with the first substitute accessorystimulation to generate physical substitute stimulation data generatedby the physical accessory device prior to transmission of the physicalsubstitute stimulation data to a virtual reality system.
 16. Thenon-transitory, machine-readable storage medium of claim 15, wherein theoperations further comprise: receiving image information from anoperational space associated with the physical accessory device; anddetermining location information associated with the physical accessorydevice and a proximate hand of a user according to the image informationthat is received, wherein the location information corresponds to thephysical substitute stimulation data.
 17. A device, comprising: aprocessing system including a processor; and a memory that storesexecutable instructions that, when executed by the processing system,facilitate performance of operations, the operations comprising:receiving actual stimulation data generated by a physical accessorydevice responsive to a user interaction with the physical accessorydevice; identifying a virtual stimulation operation of a virtualaccessory device that corresponds to virtual hand-device interactioninformation according to a mapping of a virtual hand and virtualaccessory device location information; comparing the actual stimulationdata and the virtual stimulation operation to detect a disagreement; andmodifying the virtual hand-device interaction information to generatemodified virtual hand-device interaction information responsive to thedisagreement, wherein the modified virtual hand-device interactioninformation corrects the virtual hand-device interaction information sothat an apparent location of a virtual hand-device interaction indicatedby virtual stimulation operation agrees with an actual location of theuser interaction of the physical accessory device dictated by the actualstimulation data.
 18. The device of claim 17, wherein the operationsfurther comprise: providing the modified virtual hand-device interactioninformation to a virtual reality system, wherein the modified virtualhand-device interaction information is provided by the virtual realitysystem to a virtual reality application, and wherein the virtual realityapplication superimposes a virtual hand and the virtual accessory deviceonto a virtual reality display of the virtual reality applicationaccording to the modified virtual hand-device interaction information.19. The device of claim 18, wherein the virtual reality system comprisesa virtual reality operating system.
 20. The device of claim 18, whereinthe virtual reality application comprises a virtual reality gamingapplication.